Inquiry and Model Choice
Perhaps the most useful and important, though difficult skill to learn in science is how to choose a method of inquiry (say, a model). For one thing, there is a huge array of methods to choose from, and they’ll each have their proponents and their detractors. Within that noisy environment, there is always, always a complexity bias - the fancier, more difficult, more esoteric the model, the better it must be.
For another, inquiry is faddish. In the early 1990s, everthing was “collinear.” In the early 2000s, everything suffered selection bias and needed a Heckman-style model. More recently, everything is “endogenous,” a term thrown around very loosely, with few suggestions as to remedies. Whatever the current fad is, expect Reviewer #2 to criticize your paper on those grounds, no matter how irrelevant.
How does a scholar learn to navigate this methodological minefield, to choose appropriate methods, to get persuasive and reasonable answers from their models, and persuade others they’re reasonable enough they warrant publication? I think a good starting point is to adopt and truly internalize George Box’s observation that “all models are wrong, some are useful.” The contest is not to be “right” but to make choices in a correct way that lead to a useful model.
Thinking about models as useful or not useful, say on a “usefulness” continuum, invites us to think about what makes them useful, and to reflect on what our analytic goals are. For instance, are we mainly interested in prediction? Are we primarily interested in discovery or description? Are we focused on testing hypotheses or establishing causal patterns? These are related but different sorts of inquiry - they usually imply different types of methods. A method that’s useful for one type of question may not be useful for another.
Consider a scholar interested in two principal questions. The first is “how widely influential is a variable, say age, in shaping support for political violence?” The second is “how important is this variable to explaining support for political violence compared to other variables?” These questions are not really about prediction or about causation - they’re more oriented to discovering and describing the landscape of correlations among a fairly wide set of variables across a relatively large number of studies using a relatively large number of different data sources. Pattern-seeking methods (say, ensemble models) might be really useful for these purposes.
Another scholar might be interested in what determines levels of support for political violence, when and why. This implies a different type of inquiry focused on causal paths and prediction. The methods this scholar would likely choose would be quite different from those chosen by the first scholar, perhaps using regression-based methods of causal inference.
Notice the question here is not about relative “rightness” - which scholar is “more right”. Instead, the question is absolute - is scholar #1’s inquiry or model useful? Is scholar #2’s inquiry or model useful?
What makes an inquiry “useful?” There are very involved debates on questions related to this in the philosophy of science, especially in debates about ontology, about the nature of assumptions, and about what comprises “theory.” I think about this in a very simple way. If an inquiry teaches us something and provokes further inquiry, it’s useful. The more an inquiry shows us and the more research or inquiry it provokes, the more useful it is.
Not coincidentally, the more provocative or useful a model or inquiry is, the “stupider” I feel, because I’m reminded how much I don’t know, how much more there is to know, that there are questions I didn’t know existed and new lines of inquiry to pursue far beyond what I had imagined. I love this feeling.
So what do we do? Seek out challenges - new methods, new literatures and arguments. Discuss and argue with each other. Tackle the hard problems that engage you, provoke you, challenge you, make you curious, make you feel stupid. Choose methods that are useful to your inquiry; they may be simple or complex, it doesn’t matter. What matters is that those methods or models are useful. Read a lot, especially things out of your particular (narrow) area. Your literature will become an echo chamber - avoid that by reading as broadly as you can. Be picky. Read things the challenge and engage you and interest you. Read outside the discipline - read historical and journalistic accounts of the phenomena you’re interested in.
It’s easy to be overwhelmed by methods of inquiry, but important to recognize those methods as tools in a very broad toolbox. Only some of those are useful for the questions you’re asking.
